Skip to content

[VL] Fix stoi issue when get parquet write options#11504

Merged
FelixYBW merged 1 commit intoapache:mainfrom
boneanxs:stoi_parquet_write_options
Jan 30, 2026
Merged

[VL] Fix stoi issue when get parquet write options#11504
FelixYBW merged 1 commit intoapache:mainfrom
boneanxs:stoi_parquet_write_options

Conversation

@boneanxs
Copy link
Contributor

The parquet write options could exceed INT_MAX, in such case stoi could be thrown:

Caused by: org.apache.gluten.exception.GlutenException: stoi
	at org.apache.gluten.vectorized.PlanEvaluatorJniWrapper.nativeCreateKernelWithIterator(Native Method)
	at org.apache.gluten.vectorized.NativePlanEvaluator.createKernelWithBatchIterator(NativePlanEvaluator.java:69)
	at org.apache.gluten.backendsapi.velox.VeloxIteratorApi.genFinalStageIterator(VeloxIteratorApi.scala:255)
	at org.apache.gluten.execution.WholeStageZippedPartitionsRDD.$anonfun$compute$1(WholeStageZippedPartitionsRDD.scala:59)
	at org.apache.gluten.utils.Arm$.withResource(Arm.scala:25)
	at org.apache.gluten.metrics.GlutenTimeMetric$.millis(GlutenTimeMetric.scala:37)
	at org.apache.gluten.execution.WholeStageZippedPartitionsRDD.compute(WholeStageZippedPartitionsRDD.scala:46)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:380)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:344)
	at org.apache.spark.sql.execution.VeloxColumnarWriteFilesRDD.$anonfun$compute$2(VeloxColumnarWriteFilesExec.scala:209)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1566)
	at org.apache.spark.sql.execution.VeloxColumnarWriteFilesRDD.compute(VeloxColumnarWriteFilesExec.scala:205)
	... 10 more

@github-actions github-actions bot added the VELOX label Jan 28, 2026
@FelixYBW FelixYBW merged commit f26ce78 into apache:main Jan 30, 2026
105 of 106 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants